The Wrapper Induction Environment

There is much interest in systems that automatically interact with Internet information sites. Such systems are hard to build, partly because they use hand-crafted wrappers to extract a site’s content. We advocate wrapper induction, a technique for automatically learning wrappers. Our wrapper induction e_~nvironment (WIEN) enables users quickly capture a set of example page; our wrapper learning algorithm then handles the low-level details of constructing the wrapper. Introduction. The Internet presents numerous sources of information: telephone directories, airline schedules, retail product catalogs, etc. There has been tremendous interest in informationintegration systems that automatically manipulate such sites’ content on a user’s behalf (e.g. (Etzioni & Weld 1994; Kirk et al. 1995)). Unfortunately, these sites are often formatted for people rather than machines, and no provision is made for automating the process. Specifically, the content is often embedded in an HTML page, and an information-integration system must extract the relevant text, while discarding irrelevant material such as HTML tags or advertisements. Information-integration systems typically use hand-coded wrappers to perform this information extraction process. But as the Internet grows, maintaining a large wrapper repository becomes very challenging. To simplify the wrapper-construction process, we advocate wrapper induction (Kushmerick 1997; Kushmerick, Weld, & Doorenbos 1997), a technique for automatically generating wrappers. Our wrapper induction e_n_nvironment (WIEN) helps wrapper developers rapidly gather and label the examples needed by our wrapper induction algorithm. Wrapper induction. As an example, suppose an information-integration system must extract the content shown in Fig. l(a) from the page in (b), which was rendered from the HTML in (c). The ’ccwrap’ wrapper in (d) can perform this extraction task. ’ccwrap’ operates by scanning an HTML page for particular strings (’’, ’’, etc.) that identify the parts of the page to be extracted. In a nutshell, our learning algorithms constructs wrappers like ’ccwrap’, from sets of page/label pairs such as (a)/(c). ’ccwrap’ is very simple, and most Internet site are more complicated that our fictitious example. But our empirical results indicate that our techniques are appropriate for numerous actual Internet sites: we find that 70% of surveyed sites can be handled by our techniques, and our algorithm usually requires just a handful of examples and a few CPU seconds of processing. WIEN. Fig. 1(e-i) illustrates the use of WIEN to build a wrapper for the Lycos search engine, which involves the following steps: Domain specification (e): The user can specify the attributes to be extracted. WIEN uses distinct colors to highlight the fragments of each page to be extracted for each attribute. Gathering 8J labelling examples (/): Using a standard Internet browser, the user gives WIEN a set of example pages from the site, as well as the text to be extracted from each. Using the mouse, the user drag-selects the fragments of the example page to be extracted. Building the wrapper: Once the examples have been gathered, the user simply invokes a ’Build Wrapper’ command. Once learned, the wrapper can be tested on additional examples; if it makes mistakes, then the user need only correct them and re-invoke the learner. Source mode (g): In some circumstances (e.g., when extracting URLs), the text fragments to be extracted are not rendered by the browser; WIEN handles such cases with its ’HTML source mode’. Recognizers (h): To simplify the task of labeling the examples, WIEN provides an extensible facility to automate this process. When started, WIEN loads a dynamically maintained library of recognizers. A recognizer is a procedure for examining a page and identifying "interesting" text fragments. We have built recognizers that identify URLs, email addresses, dates, times, US ZIP codes, ISBN hum131 From: AAAI Technical Report WS-98-10. Compilation copyright © 1998, AAAI (www.aaai.org). All rights reserved. (a)I <’Congo’, +242’), (’Egypt’, 20’), <’Belize’, +501’), (’Spain’, +34’) Some Country Codes Congo 242 Egypt 20 (c) Beiize 501 Spain 34 (b) (d) Congo 242 Egypt20 Belize 501 Spain 34 procedure ccwraPLR(page P) while there axe more occurrences inP of ’’ for each ((k,rk) E {(’’, ’’I,<’’, ’’)} scan in P to next occurrence of ek scan in P to next occurrence of rk return extracted <..., (country, code),...} pairs